Dockerised the package. Some more refactoring.
Objective review.
Debugging CNSRC.
Refactoring of package structure etc.
Nested JSON schemas, made form work.
SRCNet demos. Added schemas to resource directory. Added SESRC_XRD_RBD RSE.
Emailed CERN about Swiss S3 storage bucket credentials.
Added SESRC xrootd but having some issues with operations taking an age with gfal (e.g. ls of a missing file).
Helped CNSRC diagnose issue with certificates + NFS (nfsv4 needs kernel >= 5.9).
Same as yesterday.
Created metadata feature.
Started looking at creating an OIDC resource directory tool to hold e.g. provider emails, storage, SRM etc.
Number of issues with monitoring, e.g. failed transfers are repeated, skewing % success numbers.
Looked at trying to append scripted fields to data being indexed. Can either do this with runtime fields (calculated at query time) or make them indexable when a document is inserted with the following index mapping:
{
"mappings": {
"properties": {
"day_of_week": {
"type": "keyword",
"script": {
"source": "emit(doc['created_at'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT))",
"lang": "painless"
},
"index": false
},
"created_at": {
"type": "date"
},
"event_type": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"id": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"payload": {
"properties": {
"account": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"activity": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"bytes": {
"type": "long"
},
"checksum-adler": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"checksum-md5": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"child_name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"child_scope": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"child_type": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"childname": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"childscope": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"childtype": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"created_at": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"dataset": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"datasetScope": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"did_type": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"dst-rse": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"dst-rse-id": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"dst-type": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"dst-url": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"duration": {
"type": "long"
},
"expired_at": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"external-host": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"external-id": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"file-size": {
"type": "long"
},
"name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"previous-request-id": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"protocol": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"queued_at": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"reason": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"request-id": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"request-type": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"retry-count": {
"type": "long"
},
"rse": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"rule-id": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"scope": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"src-rse": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"src-rse-id": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"src-type": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"src-url": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"started_at": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"state": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"submitted_at": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"tool-id": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"transfer-endpoint": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"transfer-id": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"transfer-link": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"transfer_link": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"transferred_at": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"url": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"services": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}Either way, you can't get them out of Grafana as they are require:
{
"fields": [
"created_at",
"day_of_week"
]
}in the _search body, which isn't achievable with Lucene.
Aside: can retrieve the latest doc via _search with:
{
"size": 1,
"sort": { "created_at": "desc"},
"query": {
"match_all": {}
}
}Demo, Rucio meeting. Integrated CASRC after some initial auth/checksum issues.
Config like:
$ grep md5 /etc/xrootd/xrootd-rse-rucio.cfg
xrootd.chksum max 2 md5 adler32 crc32Here are the token bits:
$ cat /etc/xrootd/xrootd-rse-rucio.cfg
...
# <a name="-10" class="markdown-link" id="-10">Setup Auth</a>
acc.authdb /etc/xrootd/Authfile
# <a name="-11" class="markdown-link" id="-11">Token Auth</a>
ofs.authlib ++ /usr/lib64/libXrdAccSciTokens.so config=/etc/xrootd/scitokens.cfg
ofs.authorize
and the scitokens config:
$ cat /etc/xrootd/scitokens.cfg
[Global]
onmissing = passthrough
[Issuer ESCAPE]
issuer = https://iam-escape.cloud.cnaf.infn.it/
base_path = /data/
map_subject = False
default_user = <some user>$ cat /etc/xrootd/Authfile
g /escape/ska /data/ aTrying to get SM to work on k3s clusters.
Writing slides for DaCHS pres.
Looking over SM issues (not able to deploy machines with Rancher).
Creating SDC3a transfer slide.
DaCHS knowledge share.
Objectives meetings.
Finished writing up README, glossed over notes for tomorrow.
Fixed issue with docker registry certificates on rucio cluster
Looked at objectives.
Metadata roadmapping.
Got DaCHS to import data from external table & database connection.